§1.1 Field of the Invention
The present invention concerns a user interface to objects, such as informational objects for example, which are stored on, or accessed via, a machine such as a computer for example. The present invention exploits the spatial memory of people and maps one or more parameters or properties of an object to a coordinate of a location of an object.
§1.2 Related Art
A brief history of user interfaces is provided below. First, known user interfaces are introduced in §1.2.1. Then, user interfaces which facilitate information access, rather than information processing, are introduced in §1.2.2. Finally, needs of users which have not yet been met by user interfaces are listed in §1.2.3.
§1.2.1 Known User Interfaces
The way in which people interact with computing machines has evolved over the last 50 or so years. Initially, these machines were typically used as information processors, and in particular, for performing mathematical operations on numbers. People interacted with such early computing machines by punching and ordering cards to effect a sequence of commands. In later computing machines, also typically used to perform mathematical operations on numbers, people interacted with such computing machines by setting switches and viewing light emitting diodes to enter commands. With the advent of the desktop personal computer, people-machine interaction evolved from the use a keyboard to enter lines of commands, discussed in §1.2.1.1 below, to the use of a keyboard and mouse to manipulate icon metaphors of the real world, discussed in §1.2.1.2 below.
§1.2.1.1 Command Lines
Early personal computers were also used to perform mathematical operations, from engineering applications to accounting applications (e.g., spreadsheets). In addition, such early personal computers were used to enter, store, and manipulate information, such as with word processing applications for example, and to effectively access stored information, such as with relational database applications for example. People typically interacted with such computers by entering commands, in accordance with fairly rigid syntactical rules, or by entering data via a keyboard and viewing results via a video monitor. Unfortunately, since the syntactical rules for interacting with a personal computer were typically not intuitive, people would have to invest a fairly substantial amount of time to master effective interaction with the computer. Until they mastered the syntactical rules, people would often become frustrated when using computers. Casual users would often conclude that learning to interact with computers would not be worth their time. Thus, computer use was usually limited to professionals (such as accountants, engineers, and scientists for example) who needed the computational power of personal computers, and to hobbyists.
§1.2.1.2 Graphical User Interfaces
The advent of graphical user interfaces (or “GUIs”) provided a more intuitive way for people to interact with computers. The casual user no longer needed to learn syntactical rules and enter sequences of commands. As personal computers penetrated business to an ever greater degree, probably due to popular and time saving word processing and spreadsheet applications, operating systems began to employ user interfaces which used an office metaphor which included documents, folders, filing cabinets, trash cans, telephone directories, etc. These so-called desktop GUIs have served their users well over the past decade or so. However, in addition to using computers for data entry, storage, and manipulation, people are using computers for access to information to an ever increasing degree. This recent trend is discussed in §1.2.2 below.
§1.2.2 Migration from Information Processing and Desktop Metaphor to Information Access
In recent decades, and in the past five (5) to ten (10) years in particular, computers have become interconnected by networks by an ever increasing extent; initially, via local area networks (or “LANs”), and more recently via LANs, private wide area networks (or “WANs”) and the Internet. The proliferation of networks, in conjunction with the increased availability of inexpensive data storage means, has afforded computer users unprecedented access to a wealth of content. Such content may be presented to a user (or “rendered”) in the form of text, images, audio, video, etc.
The Internet is one means of inter-networking local area networks and individual computers. The popularity of the Internet has exploded in recent years. Many feel that this explosive growth was fueled by the ability to link (e.g., via Hyper-text links) resources (e.g., World Wide Web pages) so that users could seamlessly transition from various resources, even when such resources were stored at geographically remote resource servers. More specifically, the Hyper-text markup language (or “HTML”) permits documents to include hyper-text links. These hyper-text links, which are typically rendered in a text file as text in a different font or color, include network address information to related resources. More specifically, the hyper-text link has an associated uniform resource locator (or “URL”) which is an Internet address at which the linked resource is located. When a user activates a hyper-text link, for example by clicking a mouse when a displayed cursor coincides with the text associated with the hyper-text link, the related resource is accessed, downloaded, and rendered to the user. The related resource may be accessed by the same resource server that provided the previously rendered resource, or may be accessed by a geographically remote resource server. Such transiting from resource to resource, by activating hyper-text links for example, is commonly referred to as “surfing”.
Although people continue to use computers to enter information, manipulate information, and store information, in view of the foregoing developments people are using computers to access information to an ever increasing extent. In a departure from the past, the information people want to access is often not created by them (which would typically reside on the person's desktop computer), or even by a company or group to which that person belongs (which would typically reside on a storage server, accessible via a local area network). Rather, given the world wide breadth of the Internet, the information people want to access is usually created by unrelated third parties (or content providers). Unfortunately, GUIs using a desktop metaphor are not particularly well suited for such information access.
New GUIs should therefore help people find information that they want, or that they might want. Unfortunately, the very vastness of available data can overwhelm a user; desired data can become difficult to find and search heuristics employed to locate desired data often return unwanted data (also referred to as “noise”).
Various concepts have been employed to help users locate desired data. In the context of the Internet for example, some services have organized content based on a hierarchy of categories. A user may then navigate through a series of hierarchical menus to find content that may be of interest to them. An example of such a service is the YAHOO™ World Wide Web site on the Internet. Unfortunately, content, in the form of Internet “web sites” for example, must be organized by the service and users must navigate through a predetermined hierarchy of menus. If a user mistakenly believes that a category will be of interest or include what they were looking for, but the category turns out to be irrelevant, the user must backtrack through one or more hierarchical levels of categories.
Again in the context of the Internet for example, some services provide “search engines” which search databased content or “web sites” pursuant to a user query. In response to a user's query, a rank ordered list, which includes brief descriptions of the uncovered content, as well as a hypertext links (text, having associated Internet address information, which, when activated, commands a computer to retrieve content from the associated Internet address) to the uncovered content is returned. The rank ordering of the list is typically based on a match between words appearing in the query and words appearing in the content. Unfortunately, however, present limitations of search heuristics often cause irrelevant content (or “noise”) to be returned in response to a query. Again, unfortunately, the very wealth of available content impairs the efficacy of these search engines since it is difficult to separate irrelevant content from relevant content.
The foregoing means permit users to find content of interest to them, or to deliver content that may be of interest to a user. Unfortunately, neither of the foregoing means is particularly well suited when the user wants to “go back” to (or “relocate”) information (or content), to revisit that information or to revisit a familiar content provider to view new information (or content). More specifically, a person typically does not want to go through steps of navigating through a hierarchy of menus, or entering a search query to get to favored content or favored content providers.
§1.2.2.1 Information Access User Interfaces and their Limitations
In view of the shift towards using computers to access information, a number of user interfaces have been developed to help people revisit or relocate information or content providers. A sample of these user interfaces, as well as perceived limitations of such user interfaces, is presented below.
§1.2.2.1.1 Bookmarks and Favorites Lists
Some so-called “Internet browser” program services, such as Microsoft's Internet Explorer™ for example, permit people to create lists of favorite Internet locations (as located by a Uniform Resource Locator or “URL”) represented by bookmarks. Unfortunately, as the number of bookmarks in a list increases, a person's ability to find a desired bookmark becomes more difficult. Although people can arrange and organize such lists of bookmarks into hierarchies, this requires some effort (mental effort is also referred to as “cognitive load”) to organize and maintain the hierarchy. Moreover, the use of hierarchies does not fully exploit the spatial memory (This concept has also been referred to as “where it is what it is”.) of people. Finally, other than textual information regarding the information, such as a title of a web page for example, the bookmarks do not provide the user with any other information about the information.
§1.2.2.1.2 Web Book/Web Forager
The article, Card, et al., “The WebBook and Web Forager: An Information Workspace for the World-Wide Web,” Proceedings of CHI '96, pp. 111-117 (Apr. 13-18, 1996) (hereafter referred to as “the WebBook article”) also recognized the trend towards the use of computers for information access. Focusing on the use of the Internet, the WebBook article noted, among other things, that web pages are often hard to find, users get lost and have difficulty relocating pages, and users have difficulty organizing found pages. The article then discussed a WebBook™ which simulates, in three dimensions, a physical book and includes pages, each of which correspond to a web page. Hypertext links are color coded to indicate whether the referenced web page is within the currently opened book or not. If the web page referenced by the link is in the book, activating the hypertext link will effect an animation of flipping pages to the desired web page. If the web page referenced by the link is not in the book, activating the hypertext link will close the currently opened WebBook and, if the web page is in another WebBook on a simulated bookshelf, will open that WebBook to the selected web page. The user can flip or ruffle through the pages of the WebBook using various input techniques. Portions of interest on a web page may be inspected with panning and zooming operations. The Document Lens feature described in the WebBook article is related to U.S. Pat. No. 5,670,984 to Robertson (incorporated herein by reference).
Although the WebBook uses screen space efficiently and uses a familiar metaphor, it has a number of limitations. First, the book metaphor limits the number of web pages that can be seen at one time—most pages are totally occluded. Second, the book metaphor does not exploit spatial memory. That is, it is believed that user will only remember the ordered pages in a linear (or one-dimensional) manner. Finally, the WebBook does not convey any information about the properties of the information or content.
The WebBook article also discusses a Web Forager which embeds the WebBook and other objects in a hierarchical three-dimensional workspace. The workspace includes four (4) hierarchical levels; namely a focus place, an immediate memory space, a secondary storage area, and a tertiary storage area. In the focus place, objects are rendered in full size and can be acted upon by the user. The immediate memory space uses the metaphor of a desk top. The secondary storage area uses several tiers in Z space, in which objects can be moved. Finally the tertiary storage area uses a bookshelf metaphor. A book selected from the bookshelf will come up to the focus place and an object then in the focus place will be moved into the secondary memory space.
Although the Web Forager exploits, at least to some extent, spatial memory, particularly in the secondary storage area, the interaction between the focus place, immediate memory place, secondary storage area, and tertiary storage area may not be immediately obvious to new users. Further, if more than about 30 objects are placed in the secondary storage place, objects may become occluded. Finally, the use of tiers in the Z dimension of the secondary storage place limits the exploitation of a user's spatial memory.
§1.2.2.1.3 Workscape
Noting the concept of “what it is where it is”, the article, Ballay, “Designing Workscape™: An Interdisciplinary Experience”, Human Factors in Computing Systems, CHI '94, pp. 10-15 (April 1994) (hereafter referred to as “the Workscape article”), discusses a three-dimensional user interface for managing documents.
As discussed at the web page on the Internet at http://www.maya.com/Portfolio/workscape.html (downloaded Jul. 7, 1998, revised 1998), Workscape permits users to drag documents in the X-Y plane, and also push and pull documents in the Z dimension.
Although Workscape exploits spatial memory, it is apparently not concerned with the problem of object occlusion. Furthermore, it is believed that separate user operations for (i) dragging an object in the X-Y plane, and (ii) pushing and pulling documents in the Z dimension, will be cumbersome for users in practice. Finally, the location of the object does not provide a user with any information about any properties of the object.
§1.2.2.1.4 Mapa
The product MAPA, offered by Dynamic Diagrams of Providence, R.I., uses a three-dimensional display to show the organization of web pages at a web site. As a web page is selected, its children in the web site hierarchy are displayed. The pages are represented by sheets. As a user's cursor hovers over a page, a pop-up title is provided. Although MAPA may be useful for visualizing large amounts of hierarchical data, it lends itself to a predetermined hierarchies (i.e., web pages of a web site). That is, it is believed that the strength of MAPA is presenting information, not building or updating hierarchies of information.
§1.2.2.1.5 MacIntosh and Windows NT® Operating Systems
Some versions of the MacIntosh and Windows NT® Operating Systems permit users to arrange, using a mouse drag operation for example, icons on a two-dimensional surface rendered on the screen of a video monitor. Drag feedback is provided. The user's arrangement of the icons persists. However, the icons represent applications or files or directories—they are not miniature images, or thumbnails, of objects. Further, the location of the icon does not relay any information regarding any of its properties, unless the user intentionally arranges the icons in such a manner.
§1.2.2.1.6 Windows NT® 5.0
A file manager or windows explorer feature of Windows NT® 5.0 may be configured to display document or file thumbnails in a selected directory. Each thumbnail includes a title which persists (i.e., is continuously rendered). A user can rearrange these thumbnails on a display screen, for example, by using a mouse drag operation. However, no drag feedback is provided.
Furthermore, the user arrangement of the thumbnails does not persist. That is, when the selected directory is left and then reentered, the thumbnails are arranged in a default manner, not based on a previously entered user arrangement. Further, the location of the thumbnail does not relay any information regarding any of its properties, unless the user intentionally arranges the thumbnails in such a manner.
§1.2.2.1.7 InfoGrid
The article: R. Rao, S. K. Card, H. D. Jellinek, J. D. Mackinlay, and G. G. Robertson, “The Information Grid: A Framework for Information Retrieval and Retrieval-Centered Applications,” UIST '92, pp. 23-32 (Nov. 15-18, 1992) (hereafter referred to as “the InfoGrid article”) discusses a user interface for object (e.g., document) retrieval. The InfoGrid display layout includes (i) an area for entering search parameters, (ii) a control panel which includes tool buttons, (iii) a area for rendering thumbnails of objects returned from a search, (iv) an area to hold selected objects, (v) an area for depicting a search path, and (vi) an area for depicting a document and its properties. The InfoGrid does not exploit a user's spatial memory and, in fact, teaches away from using location-centric workspace retrieval. (See page 31 of the InfoGrid article.) Instead, the InfoGrid permits a user to specify an initial query by providing values for various fields in a search parameter area, after which objects may be retrieved using an iterative process. (See page 26 of the InfoGrid article.)
§1.2.2.1.8 Data Mountain
U.S. patent application Ser. No. 09/152,491 entitled “Methods, Apparatus, and Data Structures for Providing a User Interface, which Exploits Spatial Memory in Three-Dimensions, to Objects” and filed on Sep. 14, 1998 discloses a user interface in which objects, represented by thumbnails, may be positioned on a simulated three-dimensional surface. However, the position of the thumbnails on the three-dimensional surface does not necessarily convey any information about the object—the user is free to position the object in the simulated three-dimensional space. It is to be understood, however, that the user may position object thumbnails in accordance with some purposeful, or unconscious, scheme.
§1.2.3 Unmet Needs
As discussed above, there exists a need for a user interface, and in particular a graphical user interface, to information or content. Such a user interface should exploit spatial memory. For example, the user interface should permit continuous movement in the simulated space. The user interface should also be intuitive to minimize the time needed for a user to become familiar with it. For example, the user interface should not be cumbersome or require too many different types of inputs to manipulate objects. The user interface should convey, at minimal cognitive load to the user, properties or parameters of the information to the user. Finally, the user interface should provide intelligent help to the user.